Because there are many differences in real life scenes， human emotions are various in different scenes， which leads to an uneven distribution of labels in the emotion dataset. Furthermore， most traditional methods utilize model pre-training and feature engineering to enhance the expression ability of expression-related features， but do not consider the complementarity between different feature representations， which limits the generalization and robustness of the model. To address these issues， EE-GAN， an end-to-end deep learning framework including the network integration model Ens-Net was proposed. It took the characteristics of different depths and regions into consideration，the fusion of different semantic and different level features was implemented， and network integration was used to improve the learning ability of the model. Besides， facial images with specific expression labels were generated by generative adversarial network， which aimed to balance the distribution of expression labels in data augmentation. The qualitative and quantitative evaluations on CK+， FER2013 and JAFFE datasets demonstrate the effectiveness of proposed method. Compared with existing view learning methods， including Locality Preserving Projections （LPP）， EE-GAN achieves the facial expression accuracies of 82.1%， 84.8% and 91.5% on the three datasets respectively. Compared with traditional CNN models such as AlexNet， VGG， and ResNet， EE-GAN achieves the accuracy increased by at least 9 percentage points.